
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">















 
 
 
 
 
 
 
  
  


<html>
  <head>
    <script type="text/javascript" language="JavaScript">
    ORIGINAL_PAGE_PATH = "/appengine/articles/remote_api.html";
    </script>
    
    
<meta http-equiv="content-type" content="text/html; charset=utf-8"/>
<title>Accessing the datastore remotely with remote_api - Google App Engine - Google Code</title>
<script type="text/javascript"><!--
(function(){function a(){this.t={};this.tick=function(c){this.t[c]=(new Date).getTime()};this.tick("start")}var b=new a;window.jstiming={Timer:a,load:b};if(window.external&&window.external.pageT)window.jstiming.pt=window.external.pageT;})();

var _tocPath_ = '/appengine/docs/_toc.ezt';
var codesite_token = null;
//--></script>
<link href="../../css/codesite.pack.04102009.css" type="text/css" rel="stylesheet"></link>
<script src="../../js/codesite_head.pack.04102009.js" type="text/javascript"></script>
<script type="text/javascript">CODESITE_CSITimer['load'].tick('bhs');</script>
<link rel="search" type="application/opensearchdescription+xml" title="Google Code" href="http://code.google.com/osd.xml" />

<!--[if IE]><link rel="stylesheet" type="text/css" href="../../css/iehacks.css" /><![endif]-->

    <link href="../../css/semantic_headers.css" rel="stylesheet" type="text/css" />
    <link href="../css/local_extensions.css" rel="stylesheet" type="text/css" />
  </head>

  <body class="gc-documentation">

    
    
    
</a>

<div id="gb">
 <span>
  
    <a id="lang-dropdown" class="dropdown" href="http://code.google.com" onclick="return false;"><img class="globeicon" src="../../images/globe2_small.png"/><span>English</span></a>
  
 </span>
</div>

<div class="gbh" style="left: 0pt;"></div>
<div class="gbh" style="right: 0pt;"></div>


<style type="text/css">
  #gc-topnav h1 {
    padding: 0 0 0 6px;
  }
</style>


<div id="gc-container">
<a name="top"></a>
<div id="skipto">
  <a href="#gc-pagecontent-anchor">Skip to page content</a>
  <a href="#gc-toc-anchor">Skip to main navigation</a>
</div>

<div id="gc-header">
  <div id="logo"><a href="http://code.google.com">
  
  
     <img src="../../images/cleardot.gif" height="1px" width="1px" alt="Google Code Home Page" id="gc-logo-img"/>
  
  
  </a></div>
  <div id="search">
    <div id="searchForm" class="searchForm">
      <form id="cse" action="http://www.google.com/cse" accept-charset="utf-8" class="gsc-search-box" onsubmit="executeGSearch(document.getElementById('gsearchInput').value); return false;">
      <noscript>
      <input type="hidden" name="cref" value="http://code.google.com/cse/googlecode-context.xml"/>
      </noscript>
        <table class="gsc-search-box" cellpadding="0" cellspacing="0">
          <tbody>
            <tr>
              <td class="gsc-input">
                <input id="gsearchInput" type="text" name="q" maxlength="2048" class="gsc-input" autocomplete="off" title="Google Code Search" style="width:345px"/>
              </td>
              <td class="gsc-search-button">
                <div id="cs-searchresults" onclick="event.cancelBubble = true;"></div>
                <input title="Search" id="gsearchButton" class="gsc-search-button" name="sa" value="Search" type="submit"/>
              </td>
            </tr>
            <tr>
              <td colspan="2" class="greytext">e.g. "templates" or "datastore"</td>
            </tr>
          </tbody>
        </table>
      </form>
    </div> <!-- end searchForm -->
  </div> <!-- end search -->
</div> <!-- end gc-header -->


<div id="codesiteContent">

<a name="gc-topnav-anchor"></a>
<div id="gc-topnav">
  <h1>Google App Engine</h1>
  <ul id="articles" class="gc-topnav-tabs">

    <li id="home_link">
      <a href="../index.html" title="Google App Engine home page">Home</a>
    </li>
  
    <li id="docs_link">
      <a href="../docs/index.html" title="Official Google App Engine documentation">Docs</a>
    </li>
  
    <li id="faq_link">
      <a href="../kb/index.html" title="Answers to frequently asked questions about Google App Engine">FAQ</a>
    </li>
  
    <li id="articles_link">
      <a href="index.html" class="selected" title="Focused articles and tutorials for Google App Engine developers">Articles</a>
    </li>
  
    <li>
      <a href="http://googleappengine.blogspot.com/" title="Official Google App Engine blog">Blog</a>
    </li>
  
    <li>
      <a href="../community.html" title="Community home for Google App Engine">Community</a>
    </li>
  
    <li>
      <a href="../terms.html" title="Google App Engine terms of service">Terms</a>
    </li>
  
    <li>
      <a href="../downloads.html" title="Download Google App Engine">Download</a>
    </li>
  

  </ul>
</div> <!-- end gc-topnav -->

    <div class="g-section g-tpl-170">

      <a name="gc-toc-anchor"></a>  
      <div class="g-unit g-first" id="gc-toc">
        <ul>
  <li><a href="../downloads.html">Downloads</a></li>
  <li><a href="http://code.google.com/status/appengine">System Status</a></li>
  <li><a href="http://code.google.com/p/googleappengine/issues/list">Issue Tracker</a></li>
</ul>
<div class="line"></div>
<ul>
  <li><h2>Getting Started</h2>
    <ul>
      <li><a href="../docs/whatisgoogleappengine.html">What Is Google App Engine?</a></li>
      <li><a href="../docs/java/gettingstarted/index.html">Java</a>
        <ul>
              <li><a href="../docs/java/gettingstarted/introduction.html">Introduction</a></li>
    <li><a href="../docs/java/gettingstarted/installing.html">Installing the Java SDK</a></li>
    <li><a href="../docs/java/gettingstarted/creating.html">Creating a Project</a></li>
    <li><a href="../docs/java/gettingstarted/usingusers.html">Using the Users Service</a></li>
    <li><a href="../docs/java/gettingstarted/usingjsps.html">Using JSPs</a></li>
    <li><a href="../docs/java/gettingstarted/usingdatastore.html">Using the Datastore with JDO</a></li>
    <li><a href="../docs/java/gettingstarted/staticfiles.html">Using Static Files</a></li>
    <li><a href="../docs/java/gettingstarted/uploading.html">Uploading Your Application</a></li>

        </ul>
      </li>
      <li><a href="../docs/python/gettingstarted/index.html">Python</a>
        <ul>
            <li><a href="../docs/python/gettingstarted/introduction.html">Introduction</a></li>
  <li><a href="../docs/python/gettingstarted/devenvironment.html">The Development Environment</a></li>
  <li><a href="../docs/python/gettingstarted/helloworld.html">Hello, World!</a></li>
  <li><a href="../docs/python/gettingstarted/usingwebapp.html">Using the webapp Framework</a></li>
  <li><a href="../docs/python/gettingstarted/usingusers.html">Using the Users Service</a></li>
  <li><a href="../docs/python/gettingstarted/handlingforms.html">Handling Forms With webapp</a></li>
  <li><a href="../docs/python/gettingstarted/usingdatastore.html">Using the Datastore</a></li>
  <li><a href="../docs/python/gettingstarted/templates.html">Using Templates</a></li>
  <li><a href="../docs/python/gettingstarted/staticfiles.html">Using Static Files</a></li>
  <li><a href="../docs/python/gettingstarted/uploading.html">Uploading Your Application</a></li>

        </ul>
      </li>
    </ul>
  </li>
</ul>
<div class="line"></div>
<ul>
  <li><h2>Java <sup class="new">Early Look</sup></h2>
    <ul>
          <li><a href="../docs/java/overview.html">Overview</a></li>
    <li><a href="../docs/java/runtime.html">Servlet Environment</a></li>
    <li><a href="../docs/java/datastore/index.html">Storing Data</a>
      <ul>
            <li><a href="../docs/java/datastore/overview.html">Overview</a></li>
    <li><a href="../docs/java/datastore/usingjdo.html">Using JDO</a></li>
    <li><a href="../docs/java/datastore/dataclasses.html">Defining Data Classes</a></li>
    <li><a href="../docs/java/datastore/creatinggettinganddeletingdata.html">Creating, Getting and Deleting Data</a></li>
    <li><a href="../docs/java/datastore/queriesandindexes.html">Queries and Indexes</a></li>
    <li><a href="../docs/java/datastore/transactions.html">Transactions</a></li>
    <li><a href="../docs/java/datastore/relationships.html">Relationships</a></li>
    <li><a href="../docs/java/datastore/usingjpa.html">Using JPA</a></li>
    <li><a href="../docs/java/javadoc/com/google/appengine/api/datastore/package-summary.html">Low-level API</a></li>

      </ul>
    </li>
    <li><a href="../docs/java/apis.html">Services</a>
      <ul>
        <li><a href="../docs/java/memcache/index.html">Memcache</a>
          <ul>
                <li><a href="../docs/java/memcache/overview.html">Overview</a></li>
    <li><a href="../docs/java/memcache/usingjcache.html">Using JCache</a></li>
    <li><a href="../docs/java/javadoc/com/google/appengine/api/memcache/package-summary.html">Low-level API</a></li>

          </ul>
        </li>
        <li><a href="../docs/java/urlfetch/index.html">URL Fetch</a>
          <ul>
                <li><a href="../docs/java/urlfetch/overview.html">Overview</a></li>
    <li><a href="../docs/java/urlfetch/usingjavanet.html">Using java.net</a></li>
    <li><a href="../docs/java/javadoc/com/google/appengine/api/urlfetch/package-summary.html">Low-level API</a></li>

          </ul>
        </li>
        <li><a href="../docs/java/mail/index.html">Mail</a>
          <ul>
                <li><a href="../docs/java/mail/overview.html">Overview</a></li>
    <li><a href="../docs/java/mail/usingjavamail.html">Using JavaMail</a></li>
    <li><a href="../docs/java/javadoc/com/google/appengine/api/mail/package-summary.html">Low-level API</a></li>

          </ul>
        </li>
        <li><a href="../docs/java/images/index.html">Images</a>
          <ul>
                <li><a href="../docs/java/images/overview.html">Overview</a></li>
    <li><a href="../docs/java/javadoc/com/google/appengine/api/images/package-summary.html">API Reference</a></li>

          </ul>
        </li>
        <li><a href="../docs/java/users/index.html">Google Accounts</a>
          <ul>
                <li><a href="../docs/java/users/overview.html">Overview</a></li>
    <li><a href="../docs/java/javadoc/com/google/appengine/api/users/package-summary.html">API Reference</a></li>

          </ul>
        </li>
      </ul>
    </li>
    <li><a href="../docs/java/javadoc/index.html">Javadoc Reference</a></li>
    <li><a href="../docs/java/jrewhitelist.html">JRE Class White List</a></li>
    <li><a href="../docs/java/config/index.html">Configuration</a>
      <ul>
            <li><a href="../docs/java/config/webxml.html">Deployment Descriptor</a></li>
    <li><a href="../docs/java/config/appconfig.html">App Config</a></li>
    <li><a href="../docs/java/config/indexconfig.html">Index Config</a></li>
    <li><a href="../docs/java/config/cron.html">Scheduled Tasks</a></li>

      </ul>
    </li>
    <li><a href="../docs/java/tools/index.html">Tools</a>
      <ul>
            <li><a href="../docs/java/tools/devserver.html">Development Server</a></li>
    <li><a href="../docs/java/tools/uploadinganapp.html">Uploading and Managing</a></li>
    <li><a href="../docs/java/tools/eclipse.html">Google Plugin for Eclipse</a></li>
    <li><a href="../docs/java/tools/ant.html">Using Apache Ant</a></li>

      </ul>
    </li>
    <li><a href="../docs/java/howto/index.html">How-To</a>
      <ul>
              <li><a href="../docs/java/howto/unittesting.html">Unit Testing</a></li>

      </ul>
    </li>

    </ul>
  </li>
</ul>
<div class="line"></div>
<ul>
  <li><h2>Python</h2>
    <ul>
          <li><a href="../docs/python/overview.html">Overview</a></li>
    <li><a href="../docs/python/runtime.html">CGI Environment</a></li>
    <li><a href="../docs/python/datastore/index.html">Storing Data</a>
      <ul>
             <li><a href="../docs/python/datastore/overview.html">Overview</a></li>
     <li><a href="../docs/python/datastore/entitiesandmodels.html">Entities and Models</a></li>
     <li><a href="../docs/python/datastore/creatinggettinganddeletingdata.html">Creating, Getting and Deleting Data</a></li>
     <li><a href="../docs/python/datastore/keysandentitygroups.html">Keys and Entity Groups</a></li>
     <li><a href="../docs/python/datastore/queriesandindexes.html">Queries and Indexes</a></li>
     <li><a href="../docs/python/datastore/transactions.html">Transactions</a></li>
     <li><a href="../docs/python/datastore/typesandpropertyclasses.html">Types and Property Classes</a></li>
     <li><a href="../docs/python/datastore/gqlreference.html">GQL Reference</a></li>

     <li><span class="tlw-title tlw-expanded">Reference</span>
       <ul>
         <li><a href="../docs/python/datastore/modelclass.html">Model</a></li>
         <li><a href="../docs/python/datastore/expandoclass.html">Expando</a></li>
         <li><a href="../docs/python/datastore/polymodelclass.html">PolyModel</a></li>
         <li><a href="../docs/python/datastore/propertyclass.html">Property</a></li>
         <li><a href="../docs/python/datastore/queryclass.html">Query</a></li>
         <li><a href="../docs/python/datastore/gqlqueryclass.html">GqlQuery</a></li>
         <li><a href="../docs/python/datastore/keyclass.html">Key</a></li>
         <li><a href="../docs/python/datastore/functions.html">Functions</a></li>
         <li><a href="../docs/python/datastore/exceptions.html">Exceptions</a></li>
       </ul>
     </li>

      </ul>
    </li>
    <li><a href="../docs/python/apis.html">Services</a>
      <ul>
        <li><a href="../docs/python/memcache/index.html">Memcache</a>
          <ul>
                 <li><a href="../docs/python/memcache/overview.html">Overview</a></li>
      <li><a href="../docs/python/memcache/usingmemcache.html">Using Memcache</a></li>
     <li><span class="tlw-title tlw-expanded">Reference</span>
       <ul>
         <li><a href="../docs/python/memcache/clientclass.html">Client</a></li>
         <li><a href="../docs/python/memcache/functions.html">Functions</a></li>
       </ul>
     </li>

          </ul>
        </li>
        <li><a href="../docs/python/urlfetch/index.html">URL Fetch</a>
          <ul>
                 <li><a href="../docs/python/urlfetch/overview.html">Overview</a></li>
     <li><span class="tlw-title tlw-expanded">Reference</span>
       <ul>
         <li><a href="../docs/python/urlfetch/fetchfunction.html">The fetch Function</a></li>
         <li><a href="../docs/python/urlfetch/responseobjects.html">Response Objects</a></li>
         <li><a href="../docs/python/urlfetch/exceptions.html">Exceptions</a></li>
       </ul>
     </li>

          </ul>
        </li>
        <li><a href="../docs/python/mail/index.html">Mail</a>
          <ul>
                 <li><a href="../docs/python/mail/overview.html">Overview</a></li>
     <li><a href="../docs/python/mail/sendingmail.html">Sending Mail</a></li>
     <li><a href="../docs/python/mail/attachments.html">Attachments</a></li>
     <li><span class="tlw-title tlw-expanded">Reference</span>
       <ul>
         <li><a href="../docs/python/mail/emailmessageclass.html">EmailMessage</a></li>
         <li><a href="../docs/python/mail/emailmessagefields.html">Message Fields</a></li>
         <li><a href="../docs/python/mail/functions.html">Functions</a></li>
         <li><a href="../docs/python/mail/exceptions.html">Exceptions</a></li>
       </ul>
     </li>

          </ul>
        </li>
        <li><a href="../docs/python/images/index.html">Images</a>
          <ul>
                 <li><a href="../docs/python/images/overview.html">Overview</a></li>
     <li><a href="../docs/python/images/installingPIL.html">Installing PIL</a></li>
     <li><a href="../docs/python/images/usingimages.html">Using the Images API</a></li>
     <li><span class="tlw-title tlw-expanded">Reference</span>
       <ul>
         <li><a href="../docs/python/images/imageclass.html">Image</a></li>
         <li><a href="../docs/python/images/functions.html">Functions</a></li>
         <li><a href="../docs/python/images/exceptions.html">Exceptions</a></li>
       </ul>
     </li>

          </ul>
        </li>
        <li><a href="../docs/python/users/index.html">Google Accounts</a>
          <ul>
                 <li><a href="../docs/python/users/overview.html">Overview</a></li>
     <li><a href="../docs/python/users/userobjects.html">User Objects</a></li>
     <li><a href="../docs/python/users/loginurls.html">Login URLs</a></li>
     <li><a href="../docs/python/users/adminusers.html">Admin Users</a></li>

     <li><span class="tlw-title tlw-expanded">Reference</span>
       <ul>
         <li><a href="../docs/python/users/userclass.html">User</a></li>
         <li><a href="../docs/python/users/functions.html">Functions</a></li>
         <li><a href="../docs/python/users/exceptions.html">Exceptions</a></li>
       </ul>
     </li>

          </ul>
        </li>
      </ul>
    </li>
    <li><a href="../docs/python/config/index.html">Configuration</a>
      <ul>
            <li><a href="../docs/python/config/appconfig.html">App Config</a></li>
    <li><a href="../docs/python/config/indexconfig.html">Index Config</a></li>
    <li><a href="../docs/python/config/cron.html">Scheduled Tasks</a></li>

      </ul>
    </li>
    <li><a href="../docs/python/tools/index.html">Tools</a>
      <ul>
            <li><a href="../docs/python/tools/devserver.html">Development Server</a></li>
    <li><a href="../docs/python/tools/uploadinganapp.html">Uploading and Managing</a></li>
    <li><a href="../docs/python/tools/uploadingdata.html">Uploading Data</a></li>
    <li><a href="../docs/python/tools/webapp/index.html">webapp Framework</a>
      <ul>
             <li><a href="../docs/python/tools/webapp/overview.html">Overview</a></li>
     <li><a href="../docs/python/tools/webapp/running.html">Running the Application</a></li>
     <li><a href="../docs/python/tools/webapp/requesthandlers.html">Request Handlers</a></li>
     <li><a href="../docs/python/tools/webapp/requestdata.html">Request Data</a></li>
     <li><a href="../docs/python/tools/webapp/buildingtheresponse.html">Building the Response</a></li>
     <li><a href="../docs/python/tools/webapp/redirects.html">Redirects, Headers and Status Codes</a></li>
     

     <li><span class="tlw-title tlw-expanded">Reference</span>
       <ul>
         <li><a href="../docs/python/tools/webapp/requestclass.html">Request</a></li>
         <li><a href="../docs/python/tools/webapp/responseclass.html">Response</a></li>
         <li><a href="../docs/python/tools/webapp/requesthandlerclass.html">RequestHandler</a></li>
         <li><a href="../docs/python/tools/webapp/wsgiapplicationclass.html">WSGIApplication</a></li>
         <li><a href="../docs/python/tools/webapp/utilmodule.html">Utility Functions</a></li>
         
       </ul>
     </li>

      </ul>
    </li>
    <li><a href="../docs/python/tools/libraries.html">Third-party Libraries</a></li>

      </ul>
    </li>
    <li><a href="../docs/python/howto/index.html">How-To</a>
      <ul>
              <li><a href="../docs/python/howto/usinggdataservices.html">Google Data Services</a></li>

      </ul>
    </li>

    </ul>
  </li>
</ul>
<div class="line"></div>
<ul>
  <li><h2>Managing Your App</h2>
    <ul>
      <li><a href="../docs/theadminconsole.html">The Admin Console</a></li>
      <li><a href="../docs/quotas.html">Quotas</a></li>
      <li><a href="../docs/billing.html">Billing</a></li>
    </ul>
  </li>
</ul>
<div class="line"></div>
<ul>
  <li><h2>Resources</h2>
    <ul>
      <li><a href="../kb/index.html">FAQ</a></li>
      <li><a href="index.html">Articles</a></li>
      <li><a href="http://appengine-cookbook.appspot.com/">Cookbook</a></li>
      <li><a href="http://appgallery.appspot.com/">App Gallery</a></li>
      <li><a href="http://code.google.com/p/googleappengine/">SDK Code</a></li>
      <li><a href="http://code.google.com/p/google-app-engine-samples/">Sample Apps Code</a></li>
      <li><a href="../community.html">Discussion Groups</a></li>
    </ul>
  </li>
</ul>
<div class="line"></div>
<ul>
  <li><a href="../docs/roadmap.html">Product Roadmap</a></li>
  <li><a href="http://code.google.com/p/googleappengine/wiki/SdkReleaseNotes">Release Notes</a></li>
  <li><a href="../docs/revision_history.html">Revision History</a></li>
</ul>

        <a class="hidden" href="#gc-topnav-anchor">More Google App Engine resource links</a>
      </div>
      
      <a name="gc-pagecontent-anchor"></a>   
      <div class="g-unit" id="gc-pagecontent">
        <script type="text/javascript">CODESITE_docEarlyProcessing();</script>
        <h1 class="page_title">Accessing the datastore remotely with remote_api</h1>


<i>Nick Johnson</i><br>
<i>February 27, 2009</i>

<div id="jd-content">
<div class="jd-descr">
  <h2 id="introduction">
    Introduction
  </h2>
  <p>
    Often when developing or supporting an App Engine application, it is useful
    to be able to manipulate the datastore in ways that are not well suited to
    the request/response model that works so well for serving Web Applications. 
    Previously, doing these sort of operations has entailed workarounds such as
    <a href="http://code.google.com/p/app3/">app3</a> or
    <a href="http://code.google.com/p/approcket/">App Rocket</a>. Starting with
    release 1.1.9 of the App Engine SDK, however, there's a new way to interact
    with the datastore, in the form of the <code>remote_api</code> module. This
    module allows remote access to the App Engine datastore, using the same APIs
    you know and love from writing App Engine Apps.
  </p>
  <p>
    In this article, we'll introduce you to the remote_api module, describe its
    basic functionality, and show you how to get an interactive console with
    access to your app's datastore. Then, we'll give an overview of the
    limitations of the remote_api module. Finally, we'll walk through a more
    sophisticated example: an implementation of the 'Map' part of a map/reduce
    operation, allowing you to execute a function on every entity of a kind.
  </p>

  <h2 id="about">
    About the remote_api module
  </h2>
  <p>
    The remote_api module consists of two parts: A 'handler', which you install
    on the server to handle remote datastore requests, and a 'stub', which you
    set up on the client to translate datastore requests into calls to the
    remote handler. remote_api works at the lowest level of the datastore, so
    once you've set up the stub, you don't have to worry about the fact that
    you're operating on a remote datastore: With a few caveats, it works exactly
    the same as if you were accessing the datastore directly.
  </p>
  <p>
    Installing the handler is easy. Simply add the following lines to your
    <code>app.yaml</code> under the 'handlers' key:
  </p>
  <pre>handlers:
    - url: /remote_api
      script: $PYTHON_LIB/google/appengine/ext/remote_api/handler.py
      login: admin
  </pre>
  <p>
    This installs the remote_api handler under the url '<code>/remote_api</code>
    '. You can choose a different endpoint, if you wish, but you'll need to
    change the examples below. Note that the handler specifies "login: admin".
    This is extremely important, since we don't want to give just anyone
    unfettered access to our datastore!
  </p>
  <p>
    Once you've updated the <code>app.yaml</code> file, you'll need to execute
    <code>appcfg.py update</code> for your app to upload the new mapping.
  </p>
  
  <h2 id="console">
    Creating an interactive console
  </h2>
  <p>
    One of the simplest and most useful things you can do with remote_api is to
    set up an interactive Python console that has access to your production
    datastore. Create a new Python file called "appengine_console.py" and put
    the following text in it:
  </p>
  <pre class="prettyprint">#!/usr/bin/python
import code
import getpass
import sys

sys.path.append("~/google_appengine")
sys.path.append("~/google_appengine/lib/yaml/lib")

from google.appengine.ext.remote_api import remote_api_stub
from google.appengine.ext import db

def auth_func():
  return raw_input('Username:'), getpass.getpass('Password:')

if len(sys.argv) < 2:
  print "Usage: %s app_id [host]" % (sys.argv[0],)
app_id = sys.argv[1]
if len(sys.argv) > 2:
  host = sys.argv[2]
else:
  host = '%s.appspot.com' % app_id

remote_api_stub.ConfigureRemoteDatastore(app_id, '/remote_api', auth_func, host)

code.interact('App Engine interactive console for %s' % (app_id,), None, locals())
  </pre>
  <p>
    The code here should be fairly self explanatory. On line 6, we add the SDK
    directory to the Python path. The example given here will work fine if
    you're using Linux or OS X and installed the SDK in your home directory. In
    windows, change this to the path where you installed the SDK such as
    "<code>c:\google_appengine</code>". If you're using the launcher in OS X,
    you need to set the path to
    "<code>/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine</code>".
  </p>
  <p>
    On line 11, we define a function for getting credentials from the user when
    they're required. In our example, we simply prompt for them over the
    console, but this could use any method you like. Starting on line 14, we
    extract the app ID and optional hostname from the command line, and on line
    22, we use a helper function provided by remote_api to set up the
    environment to send all datastore requests to the remote datastore. Finally,
    we call code.Interact, which starts an interactive Python interpreter.
  </p>
  
  <h2 id="running">
    Running the interactive console
  </h2>
  <p>
    Since you will probably want to access modules defined by your app, such as
    the model definitions, we need to make sure that your app is in the Python
    path. The easiest way to do this is to change directory to your app's root
    directory (the one with app.yaml in it) before running our new command.
    Then, just execute:
  </p>
  <pre>python appengine_console.py myapp</pre>
  <p>
    Replace 'myapp' with the app ID of your app, and you should get a Python
    interactive console prompt.
  </p>
  <p>
    For demonstration purposes, we'll use the Guestbook app from the Getting
    Started documentation. Assuming we're in the root directory for the
    guestbook app, issue:
  </p>
  <pre class="prettyprint">>>> import helloworld
>>> from google.appengine.ext import db
  </pre>
  <p>
    Now we have access to the contents of the guestbook app, we can issue
    commands just like we would if we were writing code to run on the server:
  </p>
  <pre class="prettyprint">
>>> # Fetch the most recent 10 guestbook entries
>>> entries = helloworld.Greeting.all().order("-date").fetch(10)
>>>
>>> # Create our own guestbook entry
>>> helloworld.Greeting(content="A greeting").put()
  </pre>
  <p>
    In general, the console will act exactly as if you were accessing the
    datastore directly, but because the script is running on your own machine,
    you don't have to worry about how long it takes to run, and you can access
    all the files and resources on your local machine as you normally would!
  </p>
  
  <h2 id="limitations">
    Limitations of remote_api
  </h2>
  <p>
    The remote_api module goes to great lengths to make sure that as far as
    possible, it behaves exactly like the native App Engine datastore. In some
    cases, this means doing things that are less efficient than they might
    otherwise be. When using remote_api, here's a few things to keep in mind:
  </p>
  
  <h3 id="roundtrips">
    Every datastore request requires a round-trip
  </h3>
  <p>
    Since you're accessing the datastore over HTTP, there's a bit more overhead
    and latency than when you access it locally. In order to speed things up and
    decrease load, try to limit the number of round-trips you do by batching
    gets and puts, and fetching batches of entities from queries. This is good
    advice not just for remote_api, but for using the datastore in general,
    since a batch operation is only considered to be a single Datastore
    operation.
    For example, instead of this:
  </p>
  <pre class="prettyprint">for key in keys:
  rec = MyModel.get(key)
  rec.foo = bar
  rec.put()
  </pre>
  <p>
    you can do this:
  </p>
  <pre class="prettyprint">records = MyModel.get(keys)
for rec in records:
  rec.foo = bar
db.Put(records)
  </pre>
  <p>
    Both examples have the same effect, but the latter requires only two
    roundtrips in total, while the former requires two roundtrips for each
    entity.
  </p>
  
  <h3 id="quota">
    Requests to remote_api use quota
  </h3>
  <p>
    Since remote_api operates over HTTP, every datastore call you make incurs
    quota usage for HTTP requests, bytes in and out, as well as the usual
    datastore quota you would expect. Bear this in mind if you're using
    remote_api to do bulk updates.
  </p>
  
  <h3 id="api-limits">
    1 MB API limits apply
  </h3>
  <p>
    As when running natively, the 1MB limit on API requests and responses still
    applies. If your entities are particularly large, you may need to limit the
    number you fetch or put at a time to keep below this limit. This conflicts
    with minimising round-trips, unfortunately, so the best advice is to use the
    largest batches you can without going over the request or response size
    limitations. For most entities, this is unlikely to be an issue, however.
  </p>
  
  <h3 id="avoid-iterating">
    Avoid iterating over queries
  </h3>
  <p>
    One common pattern with datastore access is the following:
  </p>
  <pre class="prettyprint">
q = MyModel.all()
for entity in q:
  # Do something with entity
  </pre>
  <p>
    When you do this, the SDK fetches entities from the datastore in batches of
    20, fetching a new batch whenever it uses up the existing ones. Because each
    batch has to be fetched in a separate request by remote_api, it's unable to
    do this as efficiently. Instead, remote_api executes an entirely new query
    for each batch, using the offset functionality to get further into the
    results.
  </p>
  <p>
    If you know how many entities you need, you can do the whole fetch in one
    request by asking for the number you need:
  </p>
  <pre class="prettyprint">entities = MyModel.all().fetch(100)
for entity in entities:
  # Do something with entity
  </pre>
  <p>
    If you don't know how many entities you will want, you can use the __key__
    pseudo-property to efficiently iterate over large result sets. This also
    allows you to avoid the 1000 entity limit imposed on normal datastore
    queries:
  </p>
  <pre class="prettyprint">entities = MyModel.all().fetch(100)
while entities:
  for entity in entities:
    # Do something with entity
  entities = MyModel.all().filter('__key__ >', entities[-1].key()).fetch(100)
  </pre>
  
  <h3 id="transaction-efficiency">
    Transactions are less efficient
  </h3>
  <p>
    In order to implement transactions via remote_api, it accumulates
    information on entities fetched inside the transaction, along with copies of
    entities that were put or deleted inside the transaction. When the
    transaction is committed, it sends all of this information off to the App
    Engine server, where it has to fetch all the entities that were used in the
    transaction again, verify that they have not been modified, then put and
    delete all the changes the transaction made and commit it. If
    there's a conflict, the server rolls back the transaction and notifies the
    client end, which then has to repeat the process all over again.
  </p>
  <p>
    This approach works, and exactly duplicates the functionality provided by
    transactions on the local datastore, but is rather inefficient. By all means
    use transactions where they are necessary, but try to limit the number and
    complexity of the transactions you execute in the interest of efficiency.
  </p>
  
  <h2 id="work">
    Putting remote_api to work
  </h2>
  <p>
    Now that we've demonstrated the power of remote_api and outlined its
    limitations, it's time to put what we've learned to work with a practical
    tool. Frequently it would be useful to be able to iterate over every entity
    of a given kind, be it to extract their data, or to modify them and store
    the updated entities back to the datastore.
  </p>
  <p>
    In order to achieve this, we're going to implement a simple 'map' framework.
    We'll define a class, <code>Mapper</code>, that exposes a <code>map()</code>
    method for subclasses to extend, and a couple of fields - <code>KIND</code>
    and <code>FILTERS</code> - for them to define what kind to map over, and any
    filters to apply.
  </p>
  <pre class="prettyprint">class Mapper(object):
  # Subclasses should replace this with a model class (eg, model.Person).
  KIND = None

  # Subclasses can replace this with a list of (property, value) tuples to filter by.
  FILTERS = []
  
  def map(self, entity):
    """Updates a single entity.
   
    Implementers should return a tuple containing two iterables (to_update, to_delete).
    """
    return ([], [])

  def get_query(self):
    """Returns a query over the specified kind, with any appropriate filters applied."""
    q = self.KIND.all()
    for prop, value in self.FILTERS:
      q.filter("%s =" % prop, value)
    q.order("__key__")
    return q

  def run(self, batch_size=100):
    """Executes the map procedure over all matching entities."""
    q = self.get_query()
    entities = q.fetch(batch_size)
    while entities:
      to_put = []
      to_delete = []
      for entity in entities:
        map_updates, map_deletes = self.map(entity)
        to_put.extend(map_updates)
        to_delete.extend(map_deletes)
      if to_put:
        db.put(to_put)
      if to_delete:
        db.delete(to_delete)
      q = self.get_query()
      q.filter("__key__ >", entities[-1].key())
      entities = q.fetch(batch_size)
  </pre>
  <p>
    As you can see, there's not much to it. First, we define a convenience
    method, <code>get_query()</code>, that returns a query that matches the kind
    and filters specified in the class definition, sorted by the
    <code>__key__</code> pseudo-property. This method could optionally be
    overridden by a subclass, for example to support varying the filters at
    runtime, as long as it uses only equality filters, and orders the query by
    <code>__key__</code>. Then, we define an instance method, <code>run()</code>
    , which iterates over every matching entity in batches, calling the
    <code>map()</code> function on each one, and updating or deleting the entity
    as appropriate.
  </p>
  <p>
    One caveat to our Mapper: The map process does not work from a snapshot of
    the datastore. So if you return new entities from <code>map()</code> that
    themselves meet the criteria for mapping, you may get them passed in to the
    <code>map()</code> function later in the process. Whether or not they do
    depends on where their key sorts compared to the current record's key. As a
    general rule, if you're going to create new entities of the same type in a
    <code>map()</code> function, you need some way to distinguish them from the
    original entities so you don't process them a second time.
  </p>
  <p>
    In order to use this class, we define a subclass that implements the
    <code>map()</code> function. In this example, we're going to add the phrase
    'Bar!' to any guestbook entry that contains the phrase 'foo':
  </p>
  <pre class="prettyprint">class GuestbookUpdater(Mapper):
  KIND = Greeting

  def map(self, entity):
    if entity.content.lower().find('foo') != -1:
      entity.content += ' Bar!'
      return ([entity], [])
    return ([], [])
  </pre>
  <p>
    Then, we instantiate our class and call run():
  </p>
  <pre class="prettyprint"> mapper = MyMapper()
mapper.run()
  </pre>
  <p>
    You can try this out for yourself easily: Just enter the code in the 
    interactive console we set up earlier.
  </p>
  <p>
    Finally, here's a practical - though trivial - example of where our new
    framework can be useful: Deleting all the entities of a given kind.
  </p>
  <pre class="prettyprint">class MyModelDeleter(Mapper):
  KIND = MyModel

  def map(self, entity):
    return ([], [entity])
  </pre>
  <p>
    Simple! Because Mapper takes care to always access <code>KIND</code> and
    <code>FILTERS</code> as instance variables, we can even generalize this to
    allow you to select the kind and filters at runtime:
  </p>
  <pre class="prettyprint">class BulkDeleter(Mapper):
  def __init__(self, kind, filters=None):
    self.KIND = kind
    if filters:
      self.FILTERS = filters

  def map(self, entity):
    return ([], [entity])
  </pre>
  <p>
    Of course, this is only the start of what you can do with remote_api and the
    Mapper framework. If you have a novel use you've come up with, please post
    it to the group - we'd love to hear about it.
  </p>
</div>
</div>
